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Program Description 2 


Success for All® 

Success for All (SFA)® is a whole-school reform model that 
includes a reading, writing, and oral language development program 
for students in prekindergarten through eighth grade. Classroom 
reading instruction is delivered in daily 90-minute blocks to students 
grouped by reading ability. Immediate intervention with tutors who 
are certified teachers is given each day to those students who are 
having difficulty reading at the same level as their classmates. 

This intervention report focuses on the reading component 
of SFA®, which is often implemented in the context of the SFA® 


whole-school reform program. Although the whole-school reform 
program has key components that are implemented in each 
school, school sites may vary considerably in the number of 
personnel used to implement SFA®, particularly tutors and family 
support staff. The reading curricula are essentially the same at 
all schools, with each school receiving the same training, coach- 
ing support, and materials. Ratings presented in this report are 
not disaggregated by the variations in implementation of whole- 
school reforms. 


Research 3 


One study of SFA® meets the What Works Clearinghouse (WWC) 


Based on these seven studies, the WWC considers the extent 



evidence standards and six studies meet WWC evidence standards 
with reservations. Altogether, the studies included nearly 4,000 stu- 
dents attending more than 70 elementary schools across the United 
States. The seven studies focused on students in grades K-3 who 
received the SFA® intervention for varying amounts of time. 4 



of evidence for SFA® to be medium to large for alphabetics, 
comprehension, and general reading achievement. No studies 
that meet WWC evidence standards with or without reservations 
addressed fluency. 



Effectiveness Success for All® was found to have positive effects on alphabetics, mixed effects for comprehension, and potentially positive effects 
on general reading achievement. 







General reading 


Alphabetics 


Fluency Comprehension 


achievement 



Rating of effectiveness 


Positive effects 


na 


Mixed effects 


Potentially positive effects 


Improvement index 5 


Average: +13 
percentile points 
Range: 0 to +32 
percentile points 


na 


Average: +8 percentile 
points 

Range: 0 to +17 percentile 
points 


Average: +10 percentile points 
Range: +2 to +19 percentile 
points 



na = not applicable 

1 . This report has been updated to include reviews of 19 studies that have been identified since 2006 and to incorporate the results of a reexamination of the Madden et al. 
(1993) study included in the previous version of this report, described in footnote 8. Of the additional studies, 15 are not within the scope of the protocol and 4 are within 
the scope of the protocol but do not meet evidence standards. A complete list and disposition of all studies reviewed are provided in the references. 

2. The descriptive information for this program was obtained from a publicly available source: the program’s website (www.successforall.net, downloaded February 23, 2009). 
The WWC requests developers to review the program description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive 
information for this program is beyond the scope of this review. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 1 .0 (see the WWC Standards). 

4. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

5. These numbers show the average and range of improvement indices for all findings across the studies. 
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Additional program Developer and contact 

information Developed by Robert Slavin and Nancy Madden in conjunction with 
the Johns Hopkins University, Success for All® is distributed by the 
Success for All Foundation, Inc., 200 W. Towsontown Boulevard, 
Baltimore, MD 21204-5200. Email: sfainfo@successforall.org. Web: 
http://www.successforall.net. Telephone: (800) 548-4998 ext. 2372. 

Scope of use 

SFA ® is used by schools in 48 states, Guam, and the Virgin 
Islands. According to the Success for All Foundation, more than 
1,300 schools in more than 500 districts have used the SFA® 
whole-school reform program. Schools in Israel, Canada, Mexico, 
and Australia have implemented adapted versions of SFA®. 

Teaching 

During the regular daily 90-minute reading period, students 
are grouped into reading classes of 15-20 students who are all 
performing at the same reading level (regardless of age or grade 
level). Regrouping allows teachers to teach the whole class with- 
out having to break the class into multiple smaller reading groups. 

Reading teachers at every grade level begin the period by 
reading children’s literature to students. Teachers discuss the 
story with students to enhance the students’ understanding 
of the story and the story structure and to increase their listen- 
ing and speaking vocabulary. In kindergarten and first grade, 
teachers emphasize the development of language skills and 
use phonetically regular storybooks and instruction to focus on 
phonemic awareness, auditory discrimination, and sound blend- 
ing. In the second through fifth grades, teachers use school- or 
district-provided reading materials, either basal or trade books, 



in a structured set of interactive activities in which students 
read, discuss, and write about the books. At this stage, teachers 
emphasize cooperative learning activities built around partner 
reading. Students work on identifying characters, settings, and 
problem solutions in narratives. Students receive direct instruc- 
tion in reading comprehension skills. 

Teachers in their first year of teaching SFA® receive a three-day 
summer training and 12 additional on-site support days during the 
school year. Additional in-service presentations covering topics 
such as classroom management, instructional pace, and coopera- 
tive learning are made by school facilitators and other program staff 
throughout the year. Facilitators organize information sessions to 
allow teachers to share problems and solutions, suggest changes, 
and discuss individual children. Twice a year, trainers provided by 
the developer visit and observe teachers. After the first year, training 
is reinforced by regular in-services, an annual SFA® conference, 
and on-site implementation support visits for school leaders and 
teachers. The staff development model used in whole-school SFA® 
reform emphasizes relatively brief initial training with extensive 
classroom follow-up, coaching, and group discussion. 

Principals and facilitators receive five days of initial training in 
leadership, data collection and progress monitoring, classroom 
instructional practices, school climate, and intervention using 
SFA® strategies. 

Cost 

The cost of the SFA® whole-school reform program is approxi- 
mately $80,000 in the first year, about $50,000 in the second 
year, and $35,000 in the third. Teacher training and ongoing 
support are required and are included in the cost of the program. 



Research One hundred ten studies reviewed by the WWC investigated the 

effects of SFA®. One study (Borman, Slavin, Cheung, Chamberlain, 
Madden, & Chambers, 2006) is a randomized controlled trial that 
meets WWC evidence standards. Six other studies (Dianda & Fla- 
herty, 1995; Madden, Slavin, Karweit, Dolan, & Wasik, 1993; Ross, 
Alberg, & McNelis, 1997; Ross & Casey, 1998; Ross, McNelis, 



Lewis, & Loomis, 1998; and Smith, Ross, Faulks, Casey, Shapiro, 
& Johnson, 1993) are quasi-experimental designs that meet WWC 
evidence standards with reservations. The remaining 103 studies 
do not meet either WWC evidence standards or eligibility screens. 

Some studies measured the impact of SFA® after a cohort 
of students was exposed to SFA® for one, two, and three years. 
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Research (continued) To determine ratings, the WWC used results from the final year 
reported in a study for the overall domain rating, prioritizing the 
outcomes that reflected students’ exposure to the intervention 
for the longest period of time available. 6 The studies in this 
report reflect results after (1) three years of exposure to SFA® 

(2 studies), (2) two years of exposure to SFA® (2 studies), and 
(3) one year of exposure to SFA ® (3 studies). 

Meets evidence standards 

Borman et al. (2006) was a cluster randomized controlled trial 
that examined the effects of SFA® on students in grades K-2 
across 14 states. The study randomly assigned 41 schools to 
SFA® and the comparison conditions and presented findings 
on students who had completed one, two, or three years of the 
program compared with students who took part in their schools’ 
typical reading programs. The WWC based effectiveness ratings 
on findings from the third-year longitudinal sample of 1,425 
students who began the study in kindergarten in 18 intervention 
and 17 comparison schools. 

Meets evidence standards with reservations 

Dianda and Flaherty (1995) studied the impact of SFA ® on three 
different cohorts of students who started kindergarten in 1992, 
1993, or 1994. Students were from six elementary schools in 
California. Students were grouped into four language categories; 
the WWC focuses only on the English-speaking group of 539 
students for this review. 7 SFA® students were compared with 
students who did not use the SFA® program. The WWC based 
effectiveness ratings on findings for the three cohorts that were 
exposed to SFA® for two, three, or four years. 

Madden et al. (1993) evaluated the effects of SFA® in 
Baltimore City elementary schools. The study investigated the 



effects of two versions of the SFA® program: full implementation 
and dropout prevention. Although these versions varied in their 
implementation of the whole-school reform model, the reading 
curricula are essentially the same at all schools, with each 
school receiving the same training, coaching support, and mate- 
rials. Ratings presented in this report are not disaggregated by 
the variations in implementation of whole-school reforms. 8 The 
WWC based effectiveness ratings on the findings for students at 
the end of three years of implementation for the alphabetics and 
general reading achievement domains. 

Ross, Alberg, and McNelis (1997) included first-grade students 
from 19 elementary schools implementing alternative schoolwide 
programs in the Northwest. The 19 schools were formed into four 
clusters of similar schools. For this review, the WWC reported 
results from students in three SFA® schools who were compared 
with the students from three schools that implemented the Accel- 
erated Schools program. This subsample consisted of “cluster 
2A” schools, which were neither the most disadvantaged nor the 
most affluent schools in the sample. This WWC review focused on 
the findings for 252 students at the end of the second grade, who 
had received one year of the SFA® program. 

Ross and Casey (1998) examined the effects of SFA® in three 
schools in Ft. Wayne, Indiana, by comparing them with five 
schools that implemented “locally developed programs.” The 
WWC focused on students who started the program in kinder- 
garten at two SFA® schools. The WWC based effectiveness 
ratings on the findings for 288 students at the end of first grade 
who received two years of SFA®. 

Ross et al. (1998) included 97 first-grade students from 
four elementary schools located in Little Rock, Arkansas. 

Two schools that implemented the SFA® were compared with 
two matched comparison schools that did not receive the 



6. SFA ® is designed to teach children to read at grade level by third grade, and the third year of program implementation is regarded as the full “dose” of 
Success for All® (Borman et al., 2006). 

7. The WWC Beginning Reading topic focuses only on students learning to read in English (see Beginning Reading Protocol). 

8. A previous version of this report separated the two implementations. It was later determined that both used the same reading curriculum. This updated 
report does not distinguish between the two implementations. 
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Research (continued) 



Effectiveness 



intervention. The WWC based effectiveness ratings on findings 
at the end of the second grade after students received one year 
of SFA® implementation. 

Smith et al. (1993) evaluated SFA ® in two elementary schools 
in Ft. Wayne, Indiana, by comparing them with similar students 
in two matched comparison schools who did not receive SFA®. 
The WWC based effectiveness ratings on findings for 286 
students spread across kindergarten and first grade who had 
received one year of SFA® implementation. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Scheme). The extent of 
evidence takes into account the number of studies and the 
total sample size across the studies that meet WWC evidence 
standards with or without reservations. 9 

The WWC considers the extent of evidence for SFA ® to be 
medium to large for alphabetics, comprehension, and general 
reading achievement. No studies that meet WWC evidence 
standards with or without reservations addressed fluency. 



Findings 

The WWC review of interventions for Beginning Reading 
addresses student outcomes in four domains: alphabetics, flu- 
ency, comprehension, and general reading achievement. 10 The 
studies included in this report cover three domains: alphabetics, 
comprehension, and general reading achievement. Alphabetics 
includes five constructs: phonemic awareness, phonological 
awareness, print awareness, letter knowledge, and phonics. 
Comprehension includes two constructs: reading comprehen- 
sion and vocabulary development. General reading achievement 
includes outcome measures that do not explicitly differentiate 
among different reading domains (for example, a summary 
standardized test score). The findings below present the authors’ 
estimates and WWC calculated estimates of the size and the 
statistical significance of the effects on students. 11 The results 
are presented by domain for each of the SFA® studies that meets 
the WWC evidence standards with or without reservations. 



Alphabetics 

In the alphabetics domain, all seven studies addressed phonics 
outcomes and one of these studies also measured students’ 
letter knowledge skills. 

Three years of program implementation: 

• Borman et al. (2006) examined scores on the Woodcock 
Reading Mastery Test (WRMT) and reported statistically 
significant positive effects for two phonics subtests: Word 
Identification and Word Attack. The WWC analysis confirmed 
the statistical significance of these effects. 

• Madden et al. (1993) found statistically significant positive 
effects on the phonics measure (the Woodcock Language 
Proficiency Battery [WLPB] Word Attack subtest) for students 
who began receiving the intervention in preschool and statisti- 
cally significant positive effects on the WLPB Letter-Word 
Identification subtest for those who began in kindergarten. 

The WWC confirmed statistically significant positive effects on 



9. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept, external validity, such as the students’ demographics and the types of 
settings in which studies took place, are not taken into account for the categorization. 

10. For definitions of the domains, see the Beginning Reading Protocol. 

11. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of Borman et al. (2006), a correction for multiple 
comparisons was needed. In the six other studies, corrections for clustering and multiple comparisons were needed. 
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Effectiveness (continued) the phonics measure for the preschool cohort but found that 

none of the combined effects across schools for any other 
measures was statistically significant. The average effect size 
across these outcomes was substantively important accord- 
ing to WWC criteria (that is, an effect size of at least 0.25). 

Two years of program implementation: 

• Dianda and Flaherty (1995) reported effect sizes but did not 
report on the statistical significance of the effect of SFA® on 
two phonics measures: the WLPB Letter-Word Identification 
subtest and the Word Attack subtest. According to WWC 
calculations, there were no statistically significant effects of 
SFA®, but the average effect size across the two measures 
was positive and large enough to be considered substantively 
important. 

• Ross and Casey (1998) reported no statistically significant 
effect of SFA® for one phonics measure (WRMT Word Iden- 
tification subtest) but found a statistically significant positive 
effect for the other phonics measure (WRMT Word Attack 
subtest). In WWC computations, neither of the effects was 
statistically significant, and the average effect was not large 
enough to be considered substantively important. 

One year of program implementation: 

• Ross, Alberg, and McNelis (1997) did not find a statistically 
significant effect of SFA® for one phonics measure (the WRMT 
Word Identification subtest) but did find a statistically signifi- 
cant positive effect for the other phonics measure (WRMT 
Word Attack subtest). The WWC analyses showed that neither 
of the effects was statistically significant. In addition, the 
average effect size across the two outcomes was neither 
statistically significant nor large enough to be considered 
substantively important. 

• Ross et al. (1998) found no statistically significant effects of 
SFA® on the two phonics outcomes: WRMT Word Identifica- 
tion and Word Attack subtests. The WWC analyses also found 
that no effects were statistically significant, but the average 



effect size across outcomes was positive and large enough to 
be considered substantively important. 

• Smith et al. (1993) reported no statistically significant effect 
of SFA® on the letter knowledge construct (WRMT Letter 
Identification subtest) but found statistically significant posi- 
tive effects for the two phonics outcomes (WRMT Word Iden- 
tification and Word Attack subtests) for first-grade students. 
For kindergarten students, the authors found statistically 
significant positive effects for the WRMT Letter Identification 
and the Word Identification subtests. The WWC calculations 
found that although none of these effects was statistically 
significant, the average effect size across outcomes was posi- 
tive and large enough to be substantively important. 

Overall, in the alphabetics domain, two studies showed 

statistically significant positive effects. Three studies showed 
substantively important positive effects, and two studies showed 
indeterminate effects. 12 

Comprehension 

In the comprehension domain, six studies addressed reading 
comprehension outcomes, and one of these studies also mea- 
sured students’ vocabulary development skills. 

Three years of program implementation: 

• Borman et al. (2006) reported and the WWC confirmed a 
statistically significant positive effect of SFA® on the WRMT 
Passage Comprehension subtest. 

Two years of program implementation: 

• Dianda and Flaherty (1995) did not report on the statistical 
significance of the effect of SFA® on the WLPB Passage 
Comprehension subtest. The WWC found no statistically 
significant effect, but the positive effect was large enough to 
be considered substantively important according to WWC 
criteria. 

• Ross and Casey (1998) reported no statistically significant 
effect of SFA® on the WRMT Passage Comprehension 



12. Indeterminate effects are defined as effects that are not statistically significant and with effect sizes smaller than 0.25. 
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Effectiveness (continued) subtest. In addition, the WWC found that the effect size was 

positive, but not substantively important. 

One year of program implementation: 

• Ross, Alberg, and McNelis (1997) reported no statistically 
significant effect on the WRMT Passage Comprehension 
subtest. The WWC found that the effect size was positive, but 
not substantively important. 

• Ross et al. (1998) reported a positive but not statistically sig- 
nificant effect of SFA® on the WRMT Passage Comprehension 
subtest. The effect size was not large enough to be consid- 
ered substantively important according to WWC criteria. 

• Smith et al. (1993) reported no statistically significant effect 
of SFA ® on the vocabulary development measure (Peabody 
Picture Vocabulary Test) for kindergarteners. For first-graders, 
the study authors found a statistically significant positive effect 
on the WRMT Passage Comprehension subtest. The WWC 
analysis found that none of the effects was statistically signifi- 
cant; and the average effect size across all outcomes was not 
large enough to be considered substantively important. 

For the comprehension domain, one study reported a statisti- 
cally significant positive effect and had a strong design. One 
study showed substantively important positive effects, and four 
studies showed indeterminate effects. 

General reading achievement 

Six studies examined outcomes in the general reading achieve- 
ment domain. 

Three years of program implementation: 

• Dianda and Flaherty (1995) examined the effects of SFA ® on 
the combined measure of WLPB and Durrell Oral Reading 
subtest for three cohorts of students after two to four years 
of program implementation. The authors did not report on 
the statistical significance of the findings. The WWC effect 
size computations found that although none of the effects 
was statistically significant, the mean effect size across all 
outcomes was positive and large enough to be considered 
substantively important. 



• Madden et al. (1993) found statistically significant positive effects 
of SFA® on the Durrell Oral Reading subtest for students who 
began in kindergarten and first grade. The WWC computations 
found that none of the positive effects combined across schools 
was statistically significant, but the mean effect across grade lev- 
els was large enough to be considered substantively important. 

Two years of program implementation: 

• Ross and Casey (1998) reported a positive but not statistically 
significant effect of SFA® on the Durrell Oral Reading subtest. 
The effect size was not large enough to be considered sub- 
stantively important according to WWC criteria. 

One year of program implementation: 

• Ross, Alberg, and McNelis (1997) reported a positive but 
not statistically significant effect of SFA® on the Durrell Oral 
Reading subtest. The effect size was not large enough to be 
considered substantively important according to WWC criteria. 

• Smith et al. (1993) found a statistically significant positive effect 
of SFA® on the Durrell Oral Reading subtest. The WWC compu- 
tations found that the effect was not statistically significant, but 
large enough to be considered substantively important. 

• Ross et al. (1998) reported a positive but not statistically sig- 
nificant effect on the Durrell Oral Reading subtest. The effect 
size was not large enough to be considered substantively 
important according to WWC criteria. 

In the general reading domain, three studies reported sub- 
stantively important positive effects and three studies showed 
indeterminate effects. No study had a strong design. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effectiveness 
takes into account four factors: the quality of the research design, 
the statistical significance of the findings, the size of the difference 
between participants in the intervention and the comparison condi- 
tions, and the consistency in findings across studies (see the WWC 
Intervention Rating Scheme). 
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The WWC found Success 
for Alt ® to have positive 
effects in the alphabetics 
domain, potentially positive 
effects on general reading 
achievement, and mixed 
effects on comprehension 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technical 
Details of WWC-Conducted Computations). The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus the 
percentile rank of the average student in the comparison condi- 
tion. Unlike the rating of effectiveness, the improvement index is 
entirely based on the size of the effect, regardless of the statisti- 
cal significance of the effect, the study design, or the analysis. 
The improvement index can take on values between -50 and 
+50, with positive numbers denoting favorable results for the 
intervention group. 

The average improvement index for alphabetics is +13 
percentile points across the seven studies, with a range of 0 to 



+32 percentile points across findings. The average improvement 
index for comprehension is +8 percentile points across the six 
studies, with a range of 0 to +17 percentile points across find- 
ings. The average improvement index for general reading is +10 
percentile points across the six studies, with a range of +2 to +19 
percentile points across findings. 

Summary 

The WWC reviewed 110 studies on Success for All®. One of 
these studies meets WWC evidence standards; six studies meet 
WWC evidence standards with reservations; the remaining 103 
studies do not meet either WWC evidence standards or eligibility 
screens. Based on the seven studies, the WWC found positive 
effects in the alphabetics domain, potentially positive effects in 
the general reading achievement domain, and mixed effects in 
the comprehension domain. The conclusions presented in this 
report may change as new research emerges. 
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Appendix 



Appendix A1.1 


Study characteristics: Borman, Slavin, Cheung, Chamberlain, Madden, & Chambers, 2006 (randomized controlled trial) 


Characteristic 


Description 


Study citation 


Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A., Madden, N., & Chambers, B. (2006). Final reading outcomes of the national randomized field trial of Success 
for All ®. Retrieved from Success for All website: http://www.successforall.net/_images/pdfs/Third_Year_Results_06.doc. 


Participants 


The study piloted the SFA® program in fall 2001, when three schools were randomly assigned to the SFA® and three schools to the comparison condition. In fall 2002, 

35 new schools were recruited with 18 schools randomly assigned to implement SFA ® in grades K-2, and 17 schools randomly assigned to serve as comparisons. 1 The study 
presented findings after the intervention students completed one, two, and three years of the program. For the effectiveness ratings, the WWC focused on findings from the 
longitudinal sample, that is, schools and students who completed three years of the program. 2 After three years, 18 SFA® schools with 707 students and 17 comparison 
schools with 718 students remained in the longitudinal sample. 


Setting 


The analysis sample included 35 elementary schools across 14 states located in rural and small towns in the South and urban areas of the Midwest. 


Intervention 


Intervention students received the SFA® school reform program, including the SFA® reading curriculum, tutoring for students’ quarterly assessments, family support teams 
for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. Intervention schools implemented SFA® in grades K-2 and 
used their previously planned curriculum in grades 3-5. Some schools took a year to fully implement the program. 


Comparison 


Comparison schools continued using their regular, previously planned curriculum for grades K-2 (SFA® was implemented in grades 3-5). Authors conducted observations 
at all schools and indicated that there was no evidence that when SFA® was implemented in grades 3-5, students in grades K-2 were also exposed to SFA®. All sample 
students were pretested with the Peabody Picture Vocabulary Test (PPVT) prior to SFA® implementation, and schoolwide PPVT scores show equivalence between the 
program and comparison schools. Researchers also used information from the Common Core of Data (a database maintained by the National Center for Education Statistics) 
at several points over the course of the study to demonstrate the equivalence between the program and comparison schools on race/ethnicity, gender, English as a second 
language, special education, and free and reduced-price lunch. All equivalency tests were assessed at the school level, and no statistically significant differences were found. 


Primary outcomes 
and measurement 


Three subtests of the Woodcock Reading Mastery Test were administered during the period reflected in the intervention rating: Word Identification, Word Attack, and Passage 
Comprehension. 3 (See Appendices A2.1-A2.3 for more detailed descriptions of outcome measures.) 


Staff/teacher training 


SFA® teachers received three days of training during the summer and approximately eight days of on-site follow-up during the first implementation year. Success for All 
Foundation trainers visited classrooms, met with groups of teachers, looked at data on children’s progress, and provided feedback to school staff on implementation quality 
and outcomes. 



1. The 17 additional comparison schools implemented SFA® in grades 3-5, but students in grades K-2— the focus of this study and the WWC review— did not receive the intervention. 

2. The study provided analysis for two samples, the “longitudinal sample,” which included students who participated in the program for all three years, and the “in-mover sample,” which included 
the longitudinal sample plus students who transferred into the school. The WWC analysis focuses on the longitudinal sample. The WWC prioritized outcomes that reflected students’ exposure to 
the intervention for the longest period of time available. Findings reflecting students’ outcomes after shorter periods of implementation can be found in Appendices A4.1-A4.6. 

3. One additional subtest of the Woodcock Reading Mastery Test (Letter Identification) was administered during an earlier time period and is presented as an additional finding in Appendix A4.1 
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Appendix A1.2 Study characteristics: Dianda & Flaherty, 1995 (quasi-experimental design) 



Characteristic 


Description 


Study citation 


Dianda, M., & Flaherty, J. (1995, April). Effects ofSuccess for All® on the reading achievement of first graders in California bilingual programs. Paper presented at the annual 
meeting of the American Educational Research Association, San Francisco, CA. 


Participants 


This study involved seven elementary schools in California in which the majority of students were English language learners. Six schools remained by the third year of program 
implementation. Students were grouped into four language categories and received instruction in English, Spanish, or “Sheltered English.” 1 Only the English-speaking sub- 
sample was reviewed. 2 The report includes three cohorts of students who began participating in the study as kindergarteners in 1992 (99 intervention and 120 comparison 
students), 1993 (105 intervention and 62 comparison students), or 1994 (94 intervention and 59 comparison students), for a total of 539 participants. For the effectiveness 
rating, the WWC used data that reflected students’ exposure to the intervention for the longest period of time, which varied for the different cohorts and domains. 3 Exact 
attrition rates are not known for this study; however, the post-attrition intervention and comparison samples were equivalent for the English-speaking subgroup. In the overall 
sample, the percentage of students eligible for free lunch varied from 70% to 98% in intervention schools, and from 47% to 80% in comparison schools. The percentages of 
minority students were between 50% and 70% for each study condition. 


Setting 


The analysis sample included seven elementary schools in California. 


Intervention 


Intervention students received the typical SFA® curriculum, including the SFA® reading curriculum, tutoring for students, quarterly assessments, family support teams for 
students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. 


Comparison 


Comparison schools continued using their regular, previously planned curriculum. Each comparison school was matched with an SFA® school in the same district with 
students who had similar demographics and pretest scores on the Peabody Picture Vocabulary Test measure. 


Primary outcomes 
and measurement 


Three subtests of the Woodcock Language Proficiency Battery were administered: Letter-Word Identification, Word Attack, and Passage Comprehension. The authors 
presented findings from each Woodcock subtest separately and also pooled findings from the Woodcock Letter-Word Identification subtests (see Appendices A2.1-A2.3 for 
more detailed descriptions of outcome measures). 


Staff/teacher training 


SFA ® teachers received three days of training during the summer and approximately eight days of on-site follow-up during the first implementation year. Success for All 
Foundation trainers visited classrooms, met with groups of teachers, looked at data on children’s progress, and provided feedback to school staff on implementation quality 
and outcomes. Specially trained certified teachers or qualified aides worked one-to-one with the students. 



1. English language learners participate in SFA 9 in English alongside their English-dominant classmates during a common period in the morning. During the rest of the day, they receive sheltered 
content instruction or ESL instruction, depending on their level of English proficiency. 

2. The WWC Beginning Reading topic focuses only on students learning to read in English (see Beginning Reading Protocol). 

3. Findings include outcomes after two years of exposure for the alphabetics and comprehension domains, and after two (1994 cohort), three (1993 cohort), and four (1992 cohort) years of expo- 
sure for the general reading domain. Findings reflecting students’ outcomes after shorter periods of implementation can be found in Appendix A4.3. 
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Appendix A1.3 Study characteristics: Madden, Slavin, Karweit, Dolan, & Wasik, 1993 (quasi-experimental design) 



Characteristic 


Description 


Study citation 


Madden, N. A., Slavin, R. E., Karweit, N., Dolan, L., & Wasik, B. A. (1993). Success for All®-. Longitudinal effects of a restructuring program for inner-city elementary schools. 
American Educational Research Journal, 30(1), 123-148. 


Participants 


The study investigated the effects of two versions of the SFA® program: full implementation and dropout prevention. Although these versions varied in their implementation 
of the whole-school reform model, the reading curricula are essentially the same at all schools, with each school receiving the same training, coaching support, and materials. 
Ratings presented in this report are not disaggregated by the variations in implementation of whole-school reforms. Within each comparison school, one-third of the students 
were randomly selected for testing purposes. The study focused on cohorts of students who started SFA ® in prekindergarten, kindergarten, and first grade and received the 
intervention for multiple years. To determine the effectiveness ratings, the WWC focused on the latest term results available. The third-year analytic sample included 671 
students within five SFA ® schools and 671 students within five comparison schools spread across three grade levels, 1 African-American students constituted 97% to 100% of 
students in five intervention schools, with 83% to 98% of students qualified for free lunch. In comparison (Chapter 1) schools, at least 75% of students qualified for free lunch. 


Setting 


The analysis sample included 10 elementary schools in Baltimore, Maryland. 


Intervention 


Intervention students in the full implementation version received the typical SFA ® program, including the SFA ® reading curriculum, tutoring for students in grades 1-3, quar- 
terly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. Intervention students 
in the dropout prevention version had a reduced number of tutors and family support staff. Chapter 1 monies supported the dropout prevention program. 


Comparison 


The comparison condition included schools that implemented a traditional reading program built around Macmillan Connections basal series. Each comparison school was 
matched with an intervention school based on the percentage of students getting free or reduced-price lunch and historical achievement level. Students were then individually 
matched on a standardized test given by the school district. Pretest scores on WRMT Letter-Word Identification, Word Attack, and Durrell Oral Reading subtests served as 
covariates in analyses. 


Primary outcomes 
and measurement 


Two subtests of the Woodcock Language Proficiency Battery were administered: Letter-Word Identification and Word Attack. Additional measures included Durrell Analysis 
of Reading Difficulty Silent Reading and Oral Reading subtests and the California Achievement Test (CAT) Total Reading (see Appendices A2.1-A2.3 for more detailed descrip- 
tions of outcome measures). 


Staff/teacher training 


The teachers and tutors were regular certified teachers. They received detailed teacher's manuals supplemented by two to three days of in-service at the beginning of the 
school year. For teachers of grades 1-3 and for reading tutors, these training sessions focused on the implementation of the reading program. Preschool and kindergarten 
teachers and aids were trained in the use of the thematic units, and other aspects of the preschool and kindergarten models. School facilitators also organized many informa- 
tion sessions to allow teachers to share problems and solutions, suggest changes, and discuss individual children. 



1. Additional findings for a subsample of low-achieving students can be found in Appendices A4.4-A4.6. 
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Appendix A1.4 Study characteristics: Ross, Alberg, & McNelis, 1997 (quasi-experimental design) 



Characteristic 


Description 


Study citation 


Ross, S. M., Alberg, M., & McNelis, M. (1997). Evaluation of elementary school school-wide programs: Clover Park School District. Year 1: 1996-97. Memphis, TN: The 
University of Memphis, Center for Research in Education Policy. 


Participants 


The study compared whole-school improvement programs, Success for All®, Accelerated Schools, and locally developed programs, in 19 schools. Schools were divided into 
four groups based on the similarity of several school characteristics, including enrollment, percentage of minority students, percentage of students eligible for free/reduced- 
price lunch, and initial academic performance. The WWC focused on only one group, “cluster 2A,” the third highest with respect to socioeconomic status, which included 
three SFA ® schools and three Accelerated Schools, with a total number of 252 first-grade students (148 students who attended SFA® schools, 104 students who attended 
Accelerated Schools)) The study included data that reflected students’ outcomes after one year of program implementation. In the overall sample, the percentage of minority 
students in three intervention schools was between 47% and 63%. In the three comparison schools, the range was between 42% and 54%. The percentage of students 
eligible for free/reduced-price lunch varied from 63% to 66% in intervention schools, and from 66% to 71% in comparison schools. 


Setting 


The analysis sample included six elementary schools in Clover Park, Washington. 


Intervention 


Intervention students received the typical SFA® program, including the SFA® reading curriculum, tutoring for students in grades 1-3, quarterly assessments, family support 
teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. 


Comparison 


Accelerated Schools is a comprehensive school reform program that is designed to close the achievement gap between at-risk and not at-risk children. The program redesigns 
and integrates curricular, instructional, and organizational practices so that they provide enrichment for at-risk students. 


Primary outcomes 
and measurement 


Three subtests of the Woodcock Reading Mastery Test were administered: Word Identification, Word Attack, and Passage Comprehension. The Durrell Analysis of Reading 
Difficulty Oral Reading subtest was also used (see Appendices A2.1-A2.3 for more detailed descriptions of outcome measures). 


Staff/teacher training 


No information on training for the specific teachers in this study was provided. 



1. An additional group included one SFA® school and three comparison schools (one school used Accelerated Schools design, and the other two used locally developed programs), but this com- 
parison did not meet WWC evidence screens because the effect of SFA ® could not be separated from the effect of the school. 
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Appendix A1.5 Study characteristics: Ross & Casey, 1998 (quasi-experimental design) 



Characteristic 


Description 


Study citation 


Ross, S. M., & Casey, J. (1998). Longitudinal study of student literacy achievement in different Title 1 school-wide programs in Fort Wayne Community Schools year 2: First 
grade results. Memphis, TN: The University of Memphis, Center for Research in Education Policy. 


Participants 


This study examines the effects of SFA ® in two Title 1 schools by comparing them with five other Title 1 schools that were implementing locally developed schoolwide 
programs. 1 The study did not report on the initial sample size, but 288 students in kindergarten (83 students in the SFA ® schools, 205 students at comparison schools) were 
included in the final analysis sample, and the post-attrition intervention and comparison samples were equivalent on the achievement pretest measure (PPVT). The study 
included data that reflected students’ outcomes after two years of program implementation. 2 School populations ranged between 31% and 50% minority enrollment; between 
62% and 81% of students received free or reduced-price lunch. 


Setting 


The analysis sample included seven Title 1 elementary schools in Fort Wayne, Indiana. 


Intervention 


Intervention students received the typical SFA® curriculum, including the Reading Roots reading curriculum in grade 1 and the Reading Wings reading curriculum in grade 2; 
one-to-one tutoring for the lowest-achieving students by certified teacher tutors, quarterly assessments, family support teams for students’ parents, a facilitator who worked 
with school personnel, and training for all intervention teachers. 


Comparison 


The five comparison schools implemented locally developed schoolwide programs. The schools were comparable with SFA® schools on pretest PPVT measures, socio- 
economic status, and ethnicity. Four out of the five local school programs incorporated components of other branded programs, including Reading Recovery, Accelerated 
Reader, Four-Block, and STAR. These curricula place considerable emphasis on reading, use of basal readers, and multifaceted reading activities. 


Primary outcomes 
and measurement 


Three subtests of the Woodcock Reading Mastery Test were administered: Word Identification, Word Attack, and Passage Comprehension. The study presented a combined 
measure of Word Identification and Word Attack. The Durrell Analysis of Reading Difficulty Oral Reading subtest was also used (see Appendices A2.1-A2.3 for more detailed 
descriptions of outcome measures). 


Staff/teacher training 


No information on training for the specific teachers was provided in this study. 



1. The article reported on an additional intervention school that supplemented SFA® with another branded intervention ( Reading Recovery), but results from this portion of the study do not meet 
WWC evidence standards because the effect of SFA ® cannot be separated from the effect of Reading Recovery. 

2. Additional findings for a subsample of low-achieving students (that is, lowest 25% with respect to reading achievement) are reported in Appendices A4.1-A4.6. 
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Appendix A1.6 Study characteristics: Ross, McNelis, Lewis, & Loomis, 1998 (quasi-experimental design) 



Characteristic 


Description 


Study citation 


Ross, S. M., McNelis, M., Lewis, T., & Loomis, S. (1998). Evaluation of Success for All® programs: Little Rock school district year 1: 1997-1998. Memphis, TN: The University 
of Memphis, Center for Research in Education Policy. 


Participants 


This study involved 97 first-grade students with both pretest and posttest data in four schools. Two schools implemented the Success for All® program (40 students), and two 
schools were selected as their matched comparison schools (47 students). The SFA® schools and the comparison schools were similar in poverty level, achievement level, and 
enrollment. The study reported data on students’ outcomes after one year of program implementation. 


Setting 


The study took place in four elementary schools in Little Rock, Arkansas. 


Intervention 


Intervention students received the typical SFA® program, including the SFA® reading curriculum, tutoring for students in grades 1-3, quarterly assessments, family support 
teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. 


Comparison 


No information was provided on the nature of the comparison curriculum. The two comparison schools were matched to the SFA® schools based on poverty level, achieve- 
ment level, and enrollment. Pretest PPVT scores were used as a covariate to adjust for differences in students’ abilities. 


Primary outcomes 
and measurement 


Three subtests of the Woodcock Reading Mastery Test were administered: Word Identification, Word Attack, and Passage Comprehension. The Durrell Analysis of Reading 
Difficulty Oral Reading subtest was also used (see Appendices A2.1-A2.3 for more detailed descriptions of outcome measures). 


Staff/teacher training 


No information on training for the teachers in this study was provided. 
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Appendix A1.7 Study characteristics: Smith, Ross, Faulks, Casey, Shapiro, & Johnson, 1993 (quasi-experimental design) 



Characteristic 


Description 


Study citation 


Smith, L. J., Ross, S. M„ Faulks, A., Casey, J., Shapiro, M., & Johnson, B. (1993). 1991-1992 Ft. Wayne, Indiana SFA® results. Memphis, TN: The University of Memphis, 
Center for Research in Education Policy. 


Participants 


This study involved approximately 286 students in kindergarten and first grade in four elementary schools in Fort Wayne, Indiana. Two schools implemented the SFA ® 
program. Two comparison schools were matched to the intervention schools based on poverty level, historical achievement level, and ethnicity; then pairs of students were 
matched on PPVT pretest scores. There were 74 kindergarteners and 69 first-grade students in the intervention group, and 74 kindergarteners and 69 first-grade students in 
the comparison group. Exact student attrition rates are not known for this study; however, the post-attrition intervention and comparison samples were equivalent on achieve- 
ment pretest. School-level data — poverty level, achievement, and enrollment — were similar across all schools. The study included data on students' outcomes after one year 
of program implementation. 1 


Setting 


The study took place in four elementary schools in Fort Wayne, Indiana. 


Intervention 


Intervention students received the typical SFA® program, including the SFA® reading curriculum, tutoring for students, quarterly assessments, family support teams for 
students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. 


Comparison 


Comparison schools continued using their regular, previously planned curriculum. No other information was provided on the comparison curriculum. 


Primary outcomes 
and measurement 


Four subtests of the Woodcock Reading Mastery Test were used: Letter Identification, Word Identification, Word Attack, and Passage Comprehension. Additional measures 
included the Peabody Picture Vocabulary Test and Durrell Analysis of Reading Difficulty Oral Reading subtest. The Merrill Language Screening Test and the Test of Language 
Development were also administered but have not been included in this review because they were outside the scope of the Beginning Reading review (see Appendices A2.1- 
A2.3 for more detailed descriptions of outcome measures). 


Staff/teacher training 


Teachers in their first year of teaching SFA® classes received three days of summer training and two to four additional in-service days during the school year. A school facilita- 
tor monitored and provided feedback throughout the year. Twice a year, trainers provided by the developer visited and observed teachers. After the first year, training was 
reinforced by regular in-services, an annual SFA® conference, and implementation checks for the facilitators and trainers. 



1. Additional findings for a low-achieving subset of students (lowest 25% with respect to reading achievement) are presented in Appendices A4.1-A4.6. 
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Appendix A2.1 Outcome measures in the alphabetics domain by construct 



Outcome measure 


Description 


Letter knowledge 

Woodcock Reading Mastery 
Test (WRMT): Letter 
Identification subtest 


The standardized test measures the number of letters that students are able to identify correctly (Smith et al., 1993). 


Phonics 

WRMT: Word Identification 


The Word Identification subtest is a test of decoding skills. The standardized test requires the child to read aloud isolated real words that range in frequency and difficulty 



subtest (as cited in Borman et al,, 2006; Ross, Alberg, & McNelis, 1997; Ross & Casey, 1998; Ross et al., 1998; Smith et al., 1993). 

Woodcock Language Proficiency The Letter/Word Identification subtest is a standardized test that requires the child to read aloud isolated letters and real words that range in frequency and difficulty (as cited 

Battery (WLPB): Letter-Word in Dianda & Flaherty, 1995; Madden et al., 1993). 



Identification subtest 

WRMT and WLPB: Word 
Attack subtest 


The standardized test measures phonemic decoding skills by asking students to read pseudowords. Students are aware that the words are not real (as cited in Borman et al., 
2006; Dianda & Flaherty, 1995; Madden et al,, 1993; Ross, Alberg, & McNelis, 1997; Ross & Casey, 1998; Ross et al., 1998; Smith et al., 1993). 



Appendix A2.2 Outcome measures in the comprehension domain by construct 



Outcome measure 


Description 


Reading comprehension 

WRMT and WLPB: 

Passage Comprehension 
subtest 


In this standardized test, comprehension is measured by having students fill in missing words in a short paragraph (as cited in Borman et al., 2006; Dianda & Flaherty, 1995; 
Ross, Alberg, & McNelis, 1997; Ross & Casey, 1998; Ross et al., 1998; Smith et al., 1993). 


Durrell Analysis of 
Reading Difficulty (DARD): 
Silent Reading Test 


An individually administered, standardized diagnostic test that measures reading rate while students read passages silently and answer comprehension questions (as cited in 
Madden et al., 1993). 


Vocabulary development 

Peabody Picture Vocabulary 
Test (PPVT) 


A standardized, receptive vocabulary test that asks students to choose which one of four pictures corresponds to a test word spoken aloud (as cited in Smith et al., 1993). 
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Appendix A2.3 Outcome measures in the general reading domain by construct 



Outcome measure 


Description 


California Achievement Test 
(CAT) Total Reading 


A group-administered, standardized assessment battery composed of numerous reading and language-oriented subtests (as cited in Madden et at, 1993). 


DARD Oral Reading Test 


An individually administered, standardized diagnostic test that measures reading accuracy, reading rate, and oral reading comprehension (as cited in Madden et at, 1993; 
Ross, Albert, & McNelis, 1997; Ross & Casey, 1998; Ross et at, 1998; Smith et al,, 1993). 
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Appendix A3.1 Summary of findings for all domains 1 



Domain 





Alphabetics 




Comprehension 






Outcome measure 


Letter Identification 


Phonics 


Reading Comprehension 


Vocabulary Development 


General Reading 
Achievement 








Meets evidence standards 
















Borman et al, 2006 


nr 


+ 


+ 






nr 


nr 


Meets evidence standards with reservations 














Dianda & Flaherty, 1995 


nr 


(+) 


(+) 






nr 


(+) 


Madden et al, 1993 


nr 


+ 


nr 






nr 


(+) 


Ross, Alberg, & McNelis, 1997 


nr 


ind 


ind 






nr 


ind 


Ross & Casey, 1998 


nr 


ind 


ind 






nr 


ind 


Ross et al, 1998 


nr 


(+) 


ind 






nr 


ind 


Smith et al, 1993 


(+) 


(+) 


(+) 






ind 


(+) 


Rating of Effectiveness 


Positive 


Mixed Effects 


Potentially Positive 



nr = no reported outcomes under this construct 
+ = study finding was positive and statistically significant 

(+) = study finding was positive and substantively important, but not statistically significant 

ind = study finding was indeterminate, that is, neither substantively important nor statistically significant 

1. This appendix reports findings considered for the effectiveness rating and the average improvement indices in each domain. More detailed information on findings for all measures within the 
domains and the constructs that factor into the domains can be found in Appendices A3.2-A3.4. 
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Appendix A3.2 Summary of findings for alphabetics domain 1 









Authors’ findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Construct 


Sample size 
(schools/ 

Study sample 3 students) 


Success 

for All® Comparison 

group group 


Mean 

difference 4 
(SFA ®- 
comparison) 


Statistical 

Effect significance 6 

size 5 (at a = 0.05) 


Improvement 

index 7 







Borman et al., 2006 (randomized controlled trial) 6 


1 — Three years of intervention 








WRMT: Word ID 
subtest 9 


Phonics 


Kindergarten 


35/1,425 


462.96 

(23.56) 


457.41 

(25.72) 


5.55 


0.22 


Statistically 

significant 


+9 


WRMT: Word Attack 
subtest 9 


Phonics 


Kindergarten 


35/1,425 


493.43 

(16.45) 


487.73 

(17.64) 


5.70 


0.33 


Statistically 

significant 


+13 






Madden et al., 1993 (quasi-experimental design) 810 — Three years of intervention 








WLPB: Letter-Word ID 
subtest 


Phonics 


Prekindergarten 
(Cohort 1) 


10/492 


18.53 

(5.34) 


15.91 

(6.59) 


2.62 


0.44 


ns 


+17 


WLPB: Word Attack 
subtest 


Phonics 


Prekindergarten 
(Cohort 1) 


10/492 


5.46 

(4.11) 


2.25 

(3.55) 


3.21 


0.83 


Statistically 

significant 


+30 


WLPB: Letter-Word ID 
subtest 


Phonics 


Kindergarten 
(Cohort 2) 


10/440 


25.09 

(6.65) 


21.54 

(6.72) 


3.55 


0.53 


ns 


+20 


WLPB: Word Attack 
subtest 


Phonics 


Kindergarten 
(Cohort 2) 


10/440 


8.63 

(6.27) 


5.21 

(4.76) 


3.42 


0.61 


ns 


+23 


WLPB: Letter-Word ID 
subtest 


Phonics 


Grade 1 
(Cohort 3) 


10/410 


28.69 

(6.72) 


25.56 

(6.19) 


3.12 


0.48 


ns 


+19 


WLPB: Word Attack 
subtest 


Phonics 


Grade 1 
(Cohort 3) 


10/410 


10.77 

(6.94) 


7.02 

(5.49) 


3.74 


0.60 


ns 


+23 






Dianda & Flaherty, 1995 (quasi-experimental design) 8 — Two years of intervention 








WLBP: Letter-Word ID 
subtest 


Phonics 


English-speaking 
kindergarten 
(1992 cohort) 


7/219 


nr 


nr 


na 


0.34 11 


ns 


+13 


WLBP: Word Attack 
subtest 


Phonics 


English-speaking 
kindergarten 
(1992 cohort) 


7/219 


nr 


nr 


na 


0.26 11 


ns 


+10 



(continued) 
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Appendix A3.2 Summary of findings for alphabetics domain 1 (continued) 









Authors’ findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Construct 


Sample size 
(schools/ 

Study sample 3 students) 


Success 

for All® Comparison 

group group 


Mean 

difference 4 
(SFA ®- 
comparison) 


Statistical 

Effect significance 6 

size 5 (at a = 0.05) 


Improvement 

index 7 







Ross & Casey, 1998 (quasi-experimental design) 


8 — Two years of intervention 








WRMT: Word ID 
subtest 


Phonics 


Kindergarten 


7/288 


32.14 

(14.63) 


31.30 

(14.20) 


0.84 


0.06 


ns 


+2 


WRMT: Word Attack 
subtest 


Phonics 


Kindergarten 


7/288 


12.25 

(7.36) 


10.40 

(8.20) 


1.85 


0.23 


ns 


+9 






Ross, Alberg, & McNelis, 1997 (quasi-experimental design) 8 - 


—One year of intervention 








WRMT: Word ID 
subtest 


Phonics 


Grade 1 


6/252 


nr 


nr 


na 


-0.01 12 


ns 


0 


WRMT: Word Attack 
subtest 


Phonics 


Grade 1 


6/252 


18.35 


15.86 


2.49 

(8.89) 13 


0.28 12 


ns 


+11 






Ross et al., 1998 (quasi-experimental design) 8 


— One year of intervention 








WRMT: Word ID 
subtest 


Phonics 


Grade 1 


4/97 


38.27 


36.21 


2.06 

(12.31) 14 


0.17 


ns 


+7 


WRMT: Word Attack 
subtest 


Phonics 


Grade 1 


4/97 


15.17 


11.19 


3.98 

(8.89) 14 


0.44 


ns 


+17 






Smith et al., 1993 (quasi-experimental design) 1 


3 — One year of intervention 








WRMT: Word ID 
subtest 


Phonics 


Kindergarten 
(Cohort 1) 


4/148 


10.26 

(9.82) 


3.15 

(4.95) 


7.11 


0.91 


ns 


+32 


WRMT: Letter ID 
subtest 


Letter knowledge 


Kindergarten 
(Cohort 1) 


4/148 


32.43 

(4.28) 


29.36 

(7.81) 


3.07 


0.48 


ns 


+19 


WRMT: Letter ID 
subtest 


Letter knowledge 


Grade 1 
(Cohort 2) 


4/138 


nr 


nr 


na 


0.08 11 


ns 


+3 


WRMT: Word ID 
subtest 


Phonics 


Grade 1 
(Cohort 2) 


4/138 


35.04 

(10.63) 


28.00 

(14.70) 


7.04 


0.55 


ns 


+21 


WRMT: Word Attack 
subtest 


Phonics 


Grade 1 
(Cohort 2) 


4/138 


12.60 

(7.43) 


7.90 

(7.91) 


4.70 


0.61 


ns 


+23 



(continued) 
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Appendix A3.2 Summary of findings for alphabetics domain 1 (continued) 









Authors’ findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Construct 


Sample size 
(schools/ 

Study sample 3 students) 


Success 

for All® Comparison 

group group 


Mean 

difference 4 
(SFA ®- 
comparison) 


Statistical 

Effect significance 6 

size 5 (at a = 0.05) 


Improvement 

index 7 



Averages for alphabetics 15 

Borman et al., 2006 — Three years of intervention 


0.28 


Statistically 

significant 


+11 


Madden et al, 1993 — Three years of intervention 


0.58 


ns 


+22 


Dianda & Flaherty, 1995 — Two years of intervention 


0.30 


ns 


+12 


Ross & Casey, 1998 — Two years of intervention 


0.14 


ns 


+6 


Ross, Alberg, & McNelis, 1997 — One year of intervention 


0.13 


ns 


+5 


Ross et al, 1998 — One year of intervention 


0.31 


ns 


+12 


Smith et al, 1993 — One year of intervention 


0.56 


ns 


+21 


Domain average for alphabetics across all studies 


0.33 


na 


+13 


Averages by years of SFA® implementation 

Average of results from studies with three years of intervention (two studies) 


0.43 


na 


+17 


Average of results from studies with two years of intervention (two studies) 


0.22 


na 


+9 


Average of results from studies with one year of intervention (three studies) 


0.33 


na 


+13 



na = not applicable nr = not reported ns = not statistically significant 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices. Earlier findings from longitudinal studies are not included in these ratings but are 
reported in Appendix A4.1 . Subgroup findings from the studies are not included in these ratings but are reported in Appendix A4.4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calcu- 
late statistical significance. In the case of Borman et al. (2006), a correction for multiple comparisons was needed so the significance levels may differ from those reported in the original study. 
There was no need to adjust for clustering because the findings were based on HLM analyses. In the case of the six other studies, corrections for both clustering and multiple comparisons were 
needed so the significance levels may differ from those reported in the original studies. 

9. Standard deviations and adjusted means have been received through communication with the author (G. Borman, personal communication, 2006). 
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Appendix A3.2 Summary of findings for alphabetics domain 1 (continued) 

10. WWC combined means and standard deviations for the SFA® schools and for the control schools. Adjusted posttest means (with pretests standard scores as covariates) were used for effect 
size calculations. Kindergarten and grade 1 cohorts from Abbottston elementary school received four years of intervention. 

11. Authors reported effect sizes that used the comparison group standard deviation in the denominator (Glass’s delta). Effect size was computed by subtracting the comparison group mean from 
the intervention group mean and dividing the result by the comparison group standard deviation. 

12. Authors reported effect sizes adjusted for PPVT pretest scores. 

13. The WWC derived the pooled standard deviation from the reported means and effect size. 

14. Authors reported the pooled standard deviation. 

15. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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Appendix A3.3 Summary of findings for comprehension domain 1 



Outcome measure 


Construct 


Study sample 3 




Authors’ findings from the study 

Mean outcome 
(standard deviation) 2 




WWC calculations 




Sample size 
(schools/ 
students) 


Success 

for All® Comparison 

group group 


Mean 

difference 4 
(SFA ®- 
comparison) 


Effect 

size 5 


Statistical 
significance 6 
(at a = 0.05) 


Improvement 

index 7 








Borman et al., 2006 (randomized controlled trial) 8 — Three years of intervention 








WRMT: Passage 


Reading 


Kindergarten 


35/1,425 


481.41 478.33 


3.08 


0.21 


Statistically 


+8 


Comprehension subtest 9 


comprehension 






(14.20) (15.33) 






significant 








Dianda & Flaherty, 1995 (quasi-experimental design) 8 — Two years of intervention 








WLPB: Passage 


Reading 


English-speaking 


7/219 


nr nr 


na 


0.44 


ns 


+17 


Comprehension subtest 


comprehension 


kindergarten 


















(1992 cohort) 


















Ross & Casey, 1998 (quasi-experimental design) 8 — Two years of intervention 








WRMT: Passage 


Reading 


Kindergarten 


7/288 


16.09 15.40 


0.69 


0.08 


ns 


+3 


Comprehension subtest 


comprehension 






(8.46) (8.70) 














Ross, Alberg, & McNelis, 1997 (quasi-experimental design) 8 — One year of intervention 






WRMT: Passage 


Reading 


Grade 1 


6/252 


nr nr 


na 


0 . 01 11 


ns 


0 


Comprehension subtest 


comprehension 




















Ross et al., 1998 (quasi-experimental design) 8 — One year of intervention 








WRMT: Passage 


Reading 


Grade 1 


4/97 


19.19 17.73 


1.46 


0.18 


ns 


+7 


Comprehension subtest 


comprehension 








(8.1 9) 12 












Smith et al., 1993 (quasi-experimental design) 8 — One year of intervention 








Peabody Picture Vocabulary 


Vocabulary 


Kindergarten 


4/148 


nr nr 


na 


0.17 10 


ns 


+7 


Test Vocabulary 


development 


(Cohort 1) 














WRMT: Passage 


Reading 


Grade 1 


4/136 


16.37 13.91 


2.46 


0.28 


ns 


+11 


Comprehension subtest 


comprehension 


(Cohort 2) 




(8.07) (9.31) 











Averages for comprehension 13 

Borman et al., 2006 — Three years of intervention 0.21 Statistically +8 



significant 

Dianda & Flaherty, 1995 — Two years of intervention 0.44 ns +17 
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